With this files you can deploy a Kubernetes environment that includes monitoring with Prometheus, alerts to Telegram using Alert Manager and visualize metrics with Grafana.
For implementation, you need to change certain values in certain files, let's describe them.
The first file 00-prometheus-namespace.yaml, like his name indicate, only creates a namespace named monitoring (You can change its name, but you need to replace new namespace name in almost all files).
For 01-prometheus-cluster-role.yaml file, gives to Prometheus necessary permissions access to Kubernetes cluster metrics.
In file 02-prometheus-configmap.yaml has important configurations of this implementation, such as metrics name and alerts. First configuration is prometheus.rules, in this part you can add custom rules for Alert Manager, there are example rules such as High Node Memory, High Memory Usage in Pod, High Node CPU, Node Failover, etc., all based in kubernetes metrics. For more information about how this rules works, consult Prometheus Alerting documentation. Second configuration is prometheus.yml, that contains values such as scrap interval, evaluation time of rules, path location of prometheus.rules and where is allocated Alert Manager (in this case, Kubernetes service). More information in Prometheus Configuration
Using 03-prometheus-deployment.yaml can deploy Prometheus application, this file uses version v2.29.2 or higher. You need to change value <Your Time Zone> of environment variable TZ with you respective Time Zone or just delete it.
env:
- name: TZ
value: <Your Time Zone>Last file for Prometheus is 04-prometheus-svc.yaml, that is only a Kubernetes NodePort service in port 30000, you can change this exposed port or change service type.
First file 05-prometheus-alertmanager-configmap.yaml is a ConfigMap for alertmanager.yml parameters, such as timers for alerts, type of alerts and receivers. In this part, the Telegram receiver is configured to receive alerts, so that you need to create Telegram bot with BotFather, it will return your bot token. After that, create a chat group in Telegram and add your bot there, and make the following GET request https://api.telegram.org/bot<Your Bot Token>/getUpdates and you will obtain your Chat ID.
Replace <Your Telegram Chat ID> with the value you got from your bot, with everything inside the quotes. (Some Chat ID's start with a -, in this case, you must also include the - in the url).
The URL http://prometheus-bot:9087 is a Kubernetes ClusterIP service and behind it is a deployment with a Prometheus Bot that recive alerts and send to chat group in Telegram.
receivers:
- name: webhook-telegram
webhook_configs:
- send_resolved: false
url: 'http://prometheus-bot:9087/alert/<Your Telegram Chat ID>'For more information Alert Manager Configuration is your ally.
Next file, 06-prometheus-alertmanager-deployment.yaml is deployment for Alert Manager, using version v0.23.0 or higher . You need to change value <Your Time Zone> of environment variable TZ too or just delete it.
Last file, 07-prometheus-alertmanager-svc.yaml is a Kubernetes ClusterIP service, because in this case, no need to expose deployment to the world, only Prometheus can access to this service.
This part is based on prometheus_bot by inCaller, with a little changes, such as the Dockerfile.
FROM golang:1.17.1-alpine3.14 as builder
RUN apk add --no-cache git ca-certificates make tzdata
COPY . /app
RUN cd /app && \
go get -d -v && \
CGO_ENABLED=0 GOOS=linux go build -v -a -installsuffix cgo -o prometheus_bot
FROM alpine:3.13.6
COPY --from=builder /app/prometheus_bot /
RUN apk add --no-cache ca-certificates tzdata tini
RUN mkdir /etc/telegrambot/
USER nobody
EXPOSE 9087
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["/prometheus_bot","-c","/etc/telegrambot/prometheus-bot.yml","-d"]As you can see, golang and alpine version has been updated, a directory /etc/telegrambot/ has been created and flags has been added in CMD, -c for path of config file (named prometheus-bot.yml) and -d for debug. Docker image has been uploaded in Docker Hub as salvadorarreola/telegram-prometheus-bot. If you want to upload your own Docker image, clone Github repository prometheus_bot and update Dockerfile as show before (you can change directory and config file names, but you need to update configmap and deployment too).
In this file, 08-prometheus-bot-configmap.yaml, describes two config files, prometheus-bot.yml and alert-template.tmpl. First of them, prometheus-bot.yml, is for Bot Configuration options, such as template path, time zone, telegram token, etc. As you can see, you need to replace <Your Telegram bot Token>
telegram_token: "<Your Telegram bot Token>"
template_path: "/etc/telegrambot/alert-template.tmpl"
time_zone: "<Your Time Zone>"
split_token: "|"
split_msg_byte: 4096Next config file alert-template.tmpl is about what information will be send to Telegram Bot (using labels and annotations of prometheus.rules in file 08-prometheus-bot-configmap.yaml), this is a template but you need to pay attention in syntax, that is go templating language. By the way, we can use some HTML tags to further customize the message.
Template example and Telegram message received:
{{if eq .CommonLabels.alertname "High Disk Space" -}}
{{ range .Alerts }}
Alertname: <b>{{ .Labels.alertname }}</b>
Summary: <b>{{ .Annotations.summary }}</b>
Node: <b>{{ .Labels.instance }}</b>
Percentage Disk Usage: <b>{{ .Labels.value }} %</b>
Severity: <b>{{ .Labels.severity }}</b>
Status: <b>{{ .Status }}</b>
{{ end }}
{{ end -}}
Finally, we have Kubernetes Deployment 09-prometheus-bot-deployment.yaml for Telgram Bot. Like the previous ones, change value <Your Time Zone> (you can change Docker Image too if you create a new one) and a Kubernetes ClusterIP service 10-prometheus-bot-svc.yamlfor internal connection.
ConfigMap file 11-prometheus-grafana-configmap.yaml contains connection parameters to Prometheus (prometheus.yaml) using its Kubernetes Service (04-prometheus-svc.yaml). Next file is optional, home.json is a Grafana Dashboard that override default home.json in Grafana, so if Pod dies, this dashboard still be available. You can use it, create a custom or using dashboards availables in Grafana Dashboards. Dashboard used: 315 Kubernetes cluster monitoring (via Prometheus).
Next file 12-prometheus-grafana-secret.yaml is Kubernetes Secret that contains user and password for Grafana Deployment, replace <Your user in base64> and <Your password in base64>.
Next, a Kubernetes Deployment 13-prometheus-grafana-deployment.yaml, using ConfigMap to mount configuration files and Secret to set the user and password. Finally (I promise you it's the last time) replace value <Your Time Zone>.
Last file is a Kubernetes NodePort service 14-prometheus-grafana-svc.yaml to expose port 3000 of Grafana in port 31000 (or any other port that you want).
Last 5 files about kubernetes-state-metrics are use to get more useful metrics about Kubernetes cluster and health state of the ojects.